The effectiveness of information retrieval technology in electronic discovery (E-discovery) has become the subject of judicial rulings and practitioner controversy. The scale and nature of E-discovery tasks, however, has pushed traditional information retrieval evaluation approaches to their limits. This paper reviews the legal and operational context of E-discovery and the approaches to evaluating search technology that have evolved in the research community. It then describes a multi-year effort carried out as part of the Text Retrieval Conference to develop evaluation methods (...) for responsive review tasks in E-discovery. This work has led to new approaches to measuring effectiveness in both batch and interactive frameworks, large data sets, and some surprising results for the recall and precision of Boolean and statistical information retrieval methods. The paper concludes by offering some thoughts about future research in both the legal and technical communities toward the goal of reliable, effective use of information retrieval in E-discovery. (shrink)

We provide a retrospective of 25 years of the International Conference on AI and Law, which was first held in 1987. Fifty papers have been selected from the thirteen conferences and each of them is described in a short subsection individually written by one of the 24 authors. These subsections attempt to place the paper discussed in the context of the development of AI and Law, while often offering some personal reactions and reflections. As a whole, the subsections build into (...) a history of the last quarter century of the field, and provide some insights into where it has come from, where it is now, and where it might go. (shrink)

Research in Artificial Intelligence (AI) and the Law has maintained an emphasis on knowledge representation and formal reasoning during a period when statistical, data-driven approaches have ascended to dominance within AI as a whole. Electronic discovery is a legal application area, with substantial commercial and research interest, where there are compelling arguments in favor of both empirical and knowledge-based approaches. We discuss the cases for both perspectives, as well as the opportunities for beneficial synergies.